Techniques related to measuring how relevant phrases are to a document are described herein. Such measurement may be described mathematically; given a document d and a keyword k, there is a probability p(k|d), which is the probability that k is relevant to d. This measurement of keyword-document relevance can be useful for many applications, including automatically selecting online advertisements, verifying relevance of a phrase to a sponsored ad (whether on-line or off-line), verifying whether a keyword or query is relevant to a web page, or verifying whether a new keyword is relevant to a given content page.
While relevancy measurement in general is well known, there is need to measure relevancy of a phrase to a document when the phrase is not found in the document. These phrases are referred to as out-of-document phrases. Techniques described below relate to finding measures of relevance of phrases to a document when the phrases are not found in the document. The techniques may also involve measuring such relevancies in ways that allow them to be meaningfully compared with relevancies of phrases found in the document.